Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation

SPRING 2015 15 Iamar prestar aen ... “The world is changed.” In the past decade the amount of data and the scale of computation available has increased by a previously inconceivable amount. Computer science, and AI along with it, has moved solidly out of the realm of thought problems and into an empirical science. However, many of the methods we use predate this fundamental shift, including the ideal of truth. Our central purpose is to revisit this ideal in computer science and AI, expose it as a fallacy, and begin to form a new theory of truth that is more appropriate for big data semantics. We base this new theory on the claim that, outside mathematics, truth is entirely relative and is most closely related to agreement and consensus. Our theories arise from experimental data that has been published previously (Aroyo and Welty 2013a, Soberon et al. 2013, Inel et al. 2013, Dumitrache et al. 2013), and we use throughout the article examples from our natural language